Large Language Models (LLMs) have gained unprecedented prominence, achieving widespread adoption across diverse domains and integrating deeply into society. The capability to fine-tune general-purpose LLMs, such as Generative Pre-trained Transformers (GPT), for specific tasks has facilitated the emergence of numerous Custom GPTs. These tailored models are increasingly made available through dedicated marketplaces, such as OpenAI's GPT Store. However, their black-box nature introduces significant safety and compliance risks. In this work, we present a scalable framework for the automated evaluation of Custom GPTs against OpenAI's usage policies, which define the permissible behaviors of these systems. Our framework integrates three core components: (1) automated discovery and data collection of models from the GPT store, (2) a red-teaming prompt generator tailored to specific policy categories and the characteristics of each target GPT, and (3) an LLM-as-a-judge technique to analyze each prompt-response pair for potential policy violations. We validate our framework with a manually annotated ground truth, and evaluate it through a large-scale study with 782 Custom GPTs across three categories: Romantic, Cybersecurity, and Academic GPTs. Our manual annotation process achieved an F1 score of 0.975 in identifying policy violations, confirming the reliability of the framework's assessments. The results reveal that 58.7% of the analyzed models exhibit indications of non-compliance, exposing weaknesses in the GPT store's review and approval processes. Furthermore, our findings indicate that a model's popularity does not correlate with compliance, and non-compliance issues largely stem from behaviors inherited from base models rather than user-driven customizations. We believe this approach is extendable to other chatbot platforms and policy domains, improving LLM-based systems safety.

Impersonation tactics, such as app squatting and app cloning, have posed longstanding challenges in mobile app stores, where malicious actors exploit the names and reputations of popular apps to deceive users. With the rapid growth of Large Language Model (LLM) stores like GPT Store and FlowGPT, these issues have similarly surfaced, threatening the integrity of the LLM app ecosystem. In this study, we present the first large-scale analysis of LLM app squatting and cloning using our custom-built tool, LLMappCrazy. LLMappCrazy covers 14 squatting generation techniques and integrates Levenshtein distance and BERT-based semantic analysis to detect cloning by analyzing app functional similarities. Using this tool, we generated variations of the top 1000 app names and found over 5,000 squatting apps in the dataset. Additionally, we observed 3,509 squatting apps and 9,575 cloning cases across six major platforms. After sampling, we find that 18.7% of the squatting apps and 4.9% of the cloning apps exhibited malicious behavior, including phishing, malware distribution, fake content dissemination, and aggressive ad injection.

When OpenAI launched its platform for custom GPTs, Josh Brent Villocido was stoked to learn that one of his creations would be featured. The ascendant AI company announced at its November 2023 developer day that it would launch a store that would host GPTs, custom skins that run on top of its proprietary ChatGPT technology. People can create GPTs for specific tasks, like analyzing spreadsheets, drumming up tattoo designs, or providing customer support. And when OpenAI CEO Sam Altman spoke at the dev day, he touched on potential earning opportunities for developers. "Revenue sharing is important to us," Altman said." We're going to pay people who build the most useful and the most-used GPTs a portion of our revenue."

As a pivotal extension of the renowned ChatGPT, the GPT The development of Large Language Models (LLMs) has been Store serves as a dynamic marketplace for various Generative a transformative force in human life, reshaping interactions, Pre-trained Transformer (GPT) models, shaping the frontier enhancing communication, and influencing decision-making of conversational AI. This paper presents an in-depth measurement processes. A notable manifestation of this impact is ChatGPT, study of the GPT Store, with a focus on the categorization which, since its inception, has garnered widespread popularity, of GPTs by topic, factors influencing GPT popularity, evidenced by its millions of active users and its profound and the potential security risks. Our investigation starts with integration into various sectors such as education, business, assessing the categorization of GPTs in the GPT Store, analyzing and entertainment [17]. This surge in popularity not only how they are organized by topics, and evaluating the highlights the effectiveness of ChatGPT in understanding effectiveness of the classification system. We then examine and generating human-like text but also underscores the the factors that affect the popularity of specific GPTs, looking growing public interest in AI-driven solutions.

For the past few months, Morten Blichfeldt Andersen has spent many hours scouring OpenAI's GPT Store. Since it launched in January, the marketplace for bespoke bots has filled up with a deep bench of useful and sometimes quirky AI tools. Cartoon generators spin up New Yorker–style illustrations and vivid anime stills. Programming and writing assistants offer shortcuts for crafting code and prose. There's also a color analysis bot, a spider identifier, and a dating coach called RizzGPT.

Get ready to share your custom chatbot with the whole world. OpenAI recently launched its GPT Store, after it delayed the project following the chaos of CEO Sam Altman's firing and reinstatement late in 2023. Similar to OpenAI's GPT-4 model and web browsing capabilities, only those who pay 20 a month for ChatGPT Plus can create and use "GPTs." The GPT acronym in ChatGPT actually stands for "generative pretrained transformers," but in this context, the company is using GPT as a term that refers to a unique version of ChatGPT with additional parameters and a little extra training data. Here's how to make your GPT public and some advice to help you get started with the GPT Store.

It is a truth universally acknowledged, that a single man in possession of a computer must be in want of an AI girlfriend. Certainly a lot of enterprising individuals seem to think there's a lucrative market for digital romance. OpenAI recently launched its GPT Store, where paid ChatGPT users can buy and sell customized chatbots (think Apple's app store, but for chatbots) – and the offerings include a large selection of digital girlfriends. "AI girlfriend bots are already flooding OpenAI's GPT store," a headline from Quartz, who first reported on the issue, blared on Thursday. Quartz went on to note that "the AI girlfriend bots go against OpenAI's usage policy … The company bans GPTs'dedicated to fostering romantic companionship or performing regulated activities'."

ChatGPT has certainly captured the world's imagination since its release at the end of 2022. But in day-to-day life, it is still a relatively niche product--a curiosity that leads people to ask questions that begin "Have you tried …?" or "What do you think about …?" Its maker, OpenAI, has a much more expansive vision. Its aim is seemingly to completely remake how people use the internet. For that to happen, the bot needs to be more than a conversation starter: It has to be a functioning business.

OpenAI is pushing the launch of its GPT Store to early 2024, according to an email seen by The Verge. The company introduced its GPT Builder tool in early November at its first developer conference, giving subscribers an easy way to create their own custom AI bots. At the time, OpenAI also said it would soon release the GPT Store for users to list their GPTs and potentially make money from them. It was initially slated for a November launch. But, with the surprise ouster of OpenAI's since-reinstated CEO Sam Altman, the month didn't quite pan out as planned.